Toward a Robust and Universal Crowd Labeling Framework

The advent of fast and economical computers with large electronic storage has led to a large volume of data, most of which is unlabeled. While computers provide expeditious, accurate and low-cost computation, they still lag behind in many tasks that require human intelligence such as labeling medical images, videos or text. Consequently, current research focuses on a combination of computer accuracy and human intelligence to complete labeling task. In most cases labeling needs to be done by domain experts, however, because of the variability in expertise, experience, and intelligence of human beings, experts can be scarce. As an alternative to using domain experts, help is sought from non-experts, also known as Crowd, to complete tasks that cannot be readily automated. Since crowd labelers are non-expert, multiple labels per instance are acquired for quality purposes. The final label is obtained by com- bining these multiple labels. It is very common that the ground truth, instance difficulty, and the labeler ability are unknown entities. Therefore, the aggregation task becomes a “chicken and egg” problem to start with. Despite the fact that much research using machine learning and statistical techniques has been conducted in this area (e.g., [Dekel and Shamir, 2009; Hovy et al., 2013a; Liu et al., 2012; Donmez and Carbonell, 2008]), many questions remain unresolved, these include: (a) What are the best ways to evaluate labelers? (b) It is common to use expert-labeled instances (ground truth) to evaluate la- beler ability (e.g., [Le et al., 2010; Khattak and Salleb-Aouissi, 2011; Khattak and Salleb-Aouissi, 2012; Khattak and Salleb-Aouissi, 2013]). The question is, what should be the cardinality of the set of expert-labeled instances to have an accurate evaluation? (c) Which factors other than labeler expertise (e.g., difficulty of instance, prevalence of class, bias of a labeler toward a particular class) can affect the labeling accuracy? (d) Is there any optimal way to combine multiple labels to get the best labeling accuracy? (e) Should the labels provided by oppositional/malicious labelers be dis- carded and blocked? Or is there a way to use the “information” provided by oppositional/malicious labelers? (f) How can labelers and instances be evaluated if the ground truth is not known with certitude? In this thesis, we investigate these questions. We present methods that rely on few expert-labeled instances (usually 0.1% -10% of the dataset) to evaluate various parameters using a frequentist and a Bayesian approach. The estimated parameters are then used for label aggregation to produce one final label per instance. In the first part of this thesis, we propose a method called Expert Label Injected Crowd Esti- mation (ELICE) and extend it to different versions and variants. ELICE is based on a frequentist approach for estimating the underlying parameters. The first version of ELICE estimates the pa- rameters i.e., labeler expertise and data instance difficulty, using the accuracy of crowd labelers on expert-labeled instances [Khattak and Salleb-Aouissi, 2011; Khattak and Salleb-Aouissi, 2012]. The multiple labels for each instance are combined using weighted majority voting. These weights are the scores of labeler reliability on any given instance, which are obtained by inputting the pa- rameters in the logistic function. In the second version of ELICE [Khattak and Salleb-Aouissi, 2013], we introduce entropy as a way to estimate the uncertainty of labeling. This provides an advantage of differentiating between good, random and oppositional/malicious labelers. The aggregation of labels for ELICE version 2 flips the label (for binary classification) provided by the oppositional/malicious labeler thus utilizing the information that is generally discarded by other labeling methodologies. Both versions of ELICE have a cluster-based variant in which rather than making a random choice of instances from the whole dataset, clusters of data are first formed using any clustering approach e.g., K-means. Then an equal number of instances from each cluster are chosen randomly to get expert-labels. This is done to ensure equal representation of each class in the test dataset. Besides taking advantage of expert-labeled instances, the third version of ELICE [Khattak and Salleb-Aouissi, 2016], incorporates pairwise/circular comparison of labelers to labelers and in- stances to instances. The idea here is to improve accuracy by using the crowd labels, which unlike expert-labels, are available for the whole dataset and may provide a more comprehensive view of the labeler ability and instance difficulty. This is especially helpful for the case when the domain experts do not agree on one label and ground truth is not known for certain. Therefore, incorporating more information beyond expert labels can provide better results. We test the performance of ELICE on simulated labels as well as real labels obtained from Amazon Mechanical Turk. Results show that ELICE is effective as compared to state-of-the-art methods. All versions and variants of ELICE are capable of delaying phase transition. The main contribution of ELICE is that it makes the use of all possible information available from crowd and experts. Next, we also present a theoretical framework to estimate the number of expert-labeled instances needed to achieve certain labeling accuracy. Experiments are presented to demonstrate the utility of the theoretical bound. In the second part of this thesis, we present Crowd Labeling Using Bayesian Statistics (CLUBS) [Khattak and Salleb-Aouissi, 2015; Khattak et al., 2016b; Khattak et al., 2016a], a new approach for crowd labeling to estimate labeler and instance parameters along with label aggregation. Our approach is inspired by Item Response Theory (IRT). We introduce new parameters and refine the existing IRT parameters to fit the crowd labeling scenario. The main challenge is that unlike IRT, in the crowd labeling case, the ground truth is not known and has to be estimated based on the parameters. To overcome this challenge, we acquire expert-labels for a small fraction of instances in the dataset. Our model estimates the parameters based on the expert-labeled instances. The estimated parameters are used for weighted aggregation of crowd labels for the rest of the dataset. Experiments conducted on synthetic data and real datasets with heterogeneous quality crowd-labels show that our methods perform better than many state-of-the-art crowd labeling methods. We also conduct significance tests between our methods and other state-of-the-art methods to check the significance of the accuracy of these methods. The results show the superiority of our method in most cases. Moreover, we present experiments to demonstrate the impact of the accuracy of final aggregated labels when used as training data. The results essentially emphasize the need for high accuracy of the aggregated labels. In the last part of the thesis, we present past and contemporary research related to crowd la- beling. We conclude with future of crowd labeling and further research directions. To summarize, in this thesis, we have investigated different methods for estimating crowd labeling parameters and using them for label aggregation. We hope that our contribution will be useful to the crowd labeling community

Can Instruction Fine-Tuned Language Models Identify Social Bias through Prompting?

As the breadth and depth of language model applications continue to expand rapidly, it is increasingly important to build efficient frameworks for measuring and mitigating the learned or inherited social biases of these models. In this paper, we present our work on evaluating instruction fine-tuned language models' ability to identify bias through zero-shot prompting, including Chain-of-Thought (CoT) prompts. Across LLaMA and its two instruction fine-tuned versions, Alpaca 7B performs best on the bias identification task with an accuracy of 56.7%. We also demonstrate that scaling up LLM size and data diversity could lead to further performance gain. This is a work-in-progress presenting the first component of our bias mitigation framework. We will keep updating this work as we get more results

Investigation of dielectric properties and methylene intactness under multiple environmental stresses for high voltage epoxy composites

Epoxy decays its dielectric characteristics and exhibits degradation of main hydrocarbon on exposure to multiple environmental stresses. Inorganic oxides-based epoxy composites have been performingwell in many applications and short-term testing; therefore, evaluation of their dielectric and structural characteristics under extreme weathering conditions may also unleash enhancement in these characteristics. To explore dielectric properties and degradation of main hydrocarbon group, neat epoxy and silica-based epoxy microcomposite (15% micro-silica loading) and nanocomposites (5% nano-silica loading)have been prepared and subjected to acid rain, heat, ultra-violet radiation, salt fog, and humidity in a chamber that was specially fabricated in view ofservice conditions. Interesting results were obtained before and after aging. Enhanced intactness of methylene group was observed in nanocomposite followed by micro composite. Similarly, for epoxy nanocomposites not only higher dielectric constant, lower energy dissipation and conductivity was recorded before application of stresses, but also nanocomposite showed superior sustainability in these properties after aging. In all analyses, microcomposites performed better than neat epoxy but in conductivity the results of both samples were found comparable

Investigation of epoxy composites for outdoor insulation under accelerated ultraviolet exposure

Epoxy and its composites have shown remarkable properties to be used for outdoor high voltage insulation applications. However, outdoor environment that contains UV-A radiations deteriorates the organic structure of epoxy-based insulators. To analyze the surface degradation and chemical changes in structure due to UV radiations, neat epoxy (NEPOX), 15 wt.% epoxy microcomposite (EMC 15) and 5 wt.% epoxy nanocomposite (ENC 5) were prepared and subjected to a specially fabricated UV chamber under accelerated conditions. The samples were analyzed with visual inspection, STRI hydrophobicity classification, and Scanning Electron Microscopy (SEM) and Fourier Transform Infrared (FTIR) spectroscopy. The analyses confirmed the pattern of gradual degradation of surface structure due to the appearance of ester and carbonyl groups due to oxidation of hydrocarbon chains. FTIR peaks of C=O stretching at ∼1725.52 increased by 157.44%, 75.86% and 33.33% for NEPOX, EMC 15 and ENC, respectively. This shows that ENC-5 expressed better resistance against surface oxidation as compared to EMC 15 and NEPOX. Addition of fillers enhanced the chain intactness and nano filler performed even better due to greater surface area and surface energy

Investigation of Hydrothermally Stressed Silicone Rubber/Silica Micro and Nanocomposite for the Coating High Voltage Insulation Applications

Silicone rubber is a promising insulating material that has been performing well for different insulating and dielectric applications. However, in outdoor applications, environmental stresses cause structural and surface degradations that diminish its insulating properties. This effect of degradation can be reduced with the addition of a suitable filler to the polymer chains. For the investigation of structural changes and hydrophobicity four different systems were fabricated, including neat silicone rubber, a micro composite (with 15% micro-silica filler), and nanocomposites (with 2.5% and 5% nanosilica filler) by subjecting them to various hydrothermal conditions. In general, remarkable results were obtained by the addition of fillers. However, nanocomposites showed the best resistance against the applied stresses. In comparison to neat silicone rubber, the stability of the structure and hydrophobic behavior was better for micro-silica, which was further enhanced in the case of nanocomposites. The inclusion of 5% nanosilica showed the best results before and after applying aging conditions